A Data Complexity Approach to Kernel Selection for Support Vector Machines

We describe a data complexity approach to kernel selection based on the behavior of polynomial and Gaussian kernels. Our results show how the use of a Gaussian kernel produces a gram matrix with useful local information that has no equivalent counterpart in polynomial kernels. By exploiting neighborhood information embedded by data complexity measures, we are able to carry out a form of meta-generalization. Our goal is to predict which data sets are more favorable to particular kernels (Gaussian or polynomial). The end result is a framework to improve the model selection process in Support Vector Machines. Introduction Kernel methods have gained increased popularity in the machine learning community in recent years; one key strength underlying these methods is the ability to map the original training points into a higher dimensional feature space, thereby facilitating the job of a low capacity (linear) learning machine. Mapping points into high dimensional spaces has proved an effective strategy for classification in learning algorithms like Support Vector Machines (SVM), mainly because the mapping does not imply an increase in the complexity of the classifier, and because it is possible to rely on an artifice known as the kernel trick, that obviates a precise definition of the mapping functions. The recent success of kernel classification methods has prompted the design of several techniques that aim to improve performance; but only a few studies have focused on understanding the role that the kernel function plays in them. Such information is stored in the gram matrix, or kernel matrix. A kernel matrix has size n× n, where n represents the number of elements in the data set; each entry stands as the dot product of a pair of training elements mapped into a high dimensional space. Past work proposes extracting properties from this matrix to perform kernel selection (Chen et al. 2006; You, Hamsici, and Martinez 2011), using three main metrics: Fisher’s Discriminant, Bregman’s Divergence, and Homoscedasticity. Our study focuses on two of the most used kernel functions: the polynomial and the Gaussian kernels. We analyze Copyright c © 2014, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. their behavior to extract information that can be instrumental during the kernel selection process within SVM; our strategy is to attend to the relationship between kernel behavior and data complexity measures (Ho and Basu 2002), and to capture such relationship in a simple decision tree model.